autotune: keep CI search opt-in (#770)#788
Open
jhinpan wants to merge 1 commit into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds CI-safe guardrails around FlyDSL’s autotune “offline config” workflow: committed configs are validated in GPU-free unit tests, CI is explicitly documented to avoid running autotune search, and RMSNorm gains an end-to-end autotune integration test path.
Changes:
- Add a GPU-free regression test that validates every committed
configs/autotune/*.jsonis well-formed and filename/content-consistent. - Seed
configs/autotune/with an initial RMSNorm tuned config + add documentation for generating/committing configs. - Extend autotune + RMSNorm plumbing/tests to support offline emit/lookup and CI-safe behavior (no search unless explicitly enabled).
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_autotune.py | GPU-free unit tests covering autotune keying, restore/reset semantics, builder mode, and offline emit/lookup behavior. |
| tests/unit/test_autotune_configs.py | New regression guard ensuring committed offline configs under configs/autotune/ are parseable, loadable, and filename-consistent. |
| tests/kernels/test_rmsnorm_autotune.py | GPU integration tests validating RMSNorm default path, forced-search path, cache reuse, and offline emit/lookup behavior. |
| python/flydsl/autotune.py | Adds CI-safe tuning gating, hardened cache key axes, builder-mode support, restore/reset semantics, and offline config emit/lookup. |
| kernels/rmsnorm_kernel.py | Makes RMSNorm’s build-time BLOCK_THREADS knob explicit and wires known_block_size for >256 threads. |
| kernels/rmsnorm_config.py | Introduces RMSNorm default heuristic + exhaustive config space definition for tuning. |
| kernels/rmsnorm_autotune.py | Adds the autotuned RMSNorm entrypoint using builder-mode autotuning + offline key extraction. |
| docs/autotune_guide.md | New guide documenting default/search/offline modes and CI guidance (“do not tune in CI”). |
| configs/autotune/rmsnorm,N=8192,dtype=bf16,device_name=AMD_Instinct_MI350X.json | Seed committed offline tuned config for RMSNorm. |
| configs/autotune/README.md | Documents the committed-config directory purpose and generation workflow. |
| .github/workflows/flydsl.yaml | Documents that CI must not set FLYDSL_AUTOTUNE and relies on default/offline-lookup paths. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+651
to
+660
| shape_key, dtype_str = self.config_key_fn(*args, **kwargs) | ||
| expected_op = self.op_name or self._fn_name | ||
| mismatches = [] | ||
| if data.get("op") not in (None, expected_op): | ||
| mismatches.append(f"op {data.get('op')!r}!={expected_op!r}") | ||
| if data.get("dtype") not in (None, dtype_str): | ||
| mismatches.append(f"dtype {data.get('dtype')!r}!={dtype_str!r}") | ||
| if "shape" in data and dict(shape_key) != data["shape"]: | ||
| mismatches.append(f"shape {data['shape']}!={dict(shape_key)}") | ||
| if mismatches: |
Comment on lines
+320
to
+328
| # Disk cache. Prefer an explicit op_name (builder mode has fn=None); | ||
| # otherwise fall back to the wrapped fn's name. | ||
| fn_name = op_name | ||
| if fn_name is None: | ||
| fn_name = getattr(fn, "__name__", None) or getattr(fn, "func", None) | ||
| if fn_name is not None and not isinstance(fn_name, str): | ||
| fn_name = getattr(fn_name, "__name__", "unknown") | ||
| fn_name = fn_name or "unknown" | ||
| self._fn_name = fn_name |
Comment on lines
+126
to
+133
| if device_name is None: | ||
| device_name = _device_name() | ||
| parts = [_safe_component(op_name)] | ||
| for k, v in shape_key: | ||
| parts.append(f"{_safe_component(k)}={_safe_component(v)}") | ||
| parts.append(f"dtype={_safe_component(dtype_str)}") | ||
| parts.append(f"device_name={_safe_component(device_name)}") | ||
| return ",".join(parts) + ".json" |
| deterministic config with **no search**. This is the aiter/SGLang "offline" | ||
| model. See `docs/autotune_guide.md`. | ||
|
|
||
| - **Filename is the key:** `op,shape...,dtype=...,device_name=...json`. |
jhinpan
force-pushed
the
feat/autotune-ci-guard
branch
10 times, most recently
from
July 2, 2026 19:28
b5c2305 to
92406ee
Compare
6 tasks
jhinpan
force-pushed
the
feat/autotune-ci-guard
branch
from
July 21, 2026 23:55
92406ee to
61b56ce
Compare
jhinpan
force-pushed
the
feat/autotune-ci-guard
branch
from
July 21, 2026 23:56
61b56ce to
302bc84
Compare
jhinpan
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Keep broad FlyDSL test runs deterministic:
scripts/run_tests.shexplicitly setsFLYDSL_AUTOTUNE=0before pytest and standalone examples.FLYDSL_AUTOTUNE=1around the forced-search cases they own.The shared runner is used by both the source and wheel CI workflows, so this is one guard rather than duplicated workflow configuration.
Why this is the right CI boundary
Autotune search is timing-sensitive and expensive. Shared CI should verify deterministic control-flow and correctness contracts, not select or commit a performance winner from noisy runner timing.
The current offline-artifact work in #786 already owns artifact identity, validation, fallback, and emit/load tests. Its artifacts are content-addressed deployment inputs generated on the intended GPU; FlyDSL currently has no committed artifact tree. The previous
configs/autotune/registry and standalone committed-config validator were therefore both stale and unnecessary.This PR adds no job, runner, config registry, or autotuner API.
Verification
python3 -m pytest tests/unit/test_autotune.py -q— 23 passedbash scripts/check_python_style.sh --base upstream/main --include-localbash -n scripts/run_tests.shgit diff --checkRebased onto current
main; the final diff is one commit touching two files.Refs #770. Complements #786.